NLP Track at TREC-5

NLP track has been organized for the first time at TREC-5 to provide a more focused look at how NLP techniques can help in achieving better performance in information retrieval. The intent was to see if NLP techniques available today are mature enough to have an impact on IR, specifically if and when they can offer an advantage over purely quantitative methods. This was also a place to try some more expensive and more risky solutions than those used in main TREC evaluations. 1. AIMS More specifically, there were two principal aims of NLP track evaluations: 1. To see whether NLP has value in specific retrieval circumstances even if it has not hitherto been proven advantageous for routine document/text indexing and retrieval. 2. To see if NLP can be effectively used as a means to translate an NL text into whatever representation the search engine allows: this applies to either documents or queries, or both. In termbased systems, we have a representation that is basically: terms + weights + “=” (i.e., equivalence relation between terms). Can NLP help to get closer to the ‘optimal’ query. 2. PARTICIPANTS Five teams participated in this NLP track: GE/Rutgers/NYU/Lockheed Martin, Xerox, Mitre, Claritech, and ISS Singapore. Results were submitted by the first four teams only. In addition, Chris Buckley supplied baselines for Sabir/SMART system. Other “baselines” were created by GE abd Xerox teams running their system in no-NLP mode.